When an image or scene is captured on a camera or provided on some other electronic device or computer as a digital image, it can be desirable to modify the image in ways that require the device to first segment the foreground of the image from the background. For example, a user may want to change the background in the image for entertainment reasons, practical reasons such as to replace the background of a person speaking in a video conference to provide a background more appropriate or less distracting for business purposes, or artistic reasons. The background-foreground segmentation also may be used for computer vision, object recognition, medical imaging, video coding efficiency, and other reasons. Depending on the application and type of camera used to capture the images, three-dimensional data in the form of depth maps may be used to generate the segmentation and establish a three-dimensional space for the image and to be used by other applications.
Often times the segmentation that uses three dimensional data may result in shadows formed near the border between the segments that is missing depth data and may include distortions and artifacts as well as discoloration that may result in objects entirely hidden within the shadows that should be visible if a scene of real objects was viewed by a person in the real world. This may be caused by a number of reasons. First, the perspective of an image may be shifted from the perspective used to form depth values for an original image, and may first be formed from the RGB pixel values. Depth values are then warped (depth to color warping) onto the RGB image, and are not always applied correctly so that shadow areas without depth data may be formed on such an image. Otherwise, areas of a scene may not be in the visible line of sight from one of two cameras in a 3D stereo camera system, or either the sensor or projector in an IR projection system. Also, very dark texture areas absorb IR commonly emitted from a depth projector. In these three cases, shadow areas with no depth data may be developed. The conventional solution is to fill the shadow areas with the background depth values, but this is often inaccurate when the correct depth value cannot be determined, and it does not reveal objects or texture that may be hidden in the shadows.